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Motivation of Boosting 
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(Class): Apples are circular. ] 
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e students: simple hypotheses g; (like vertical/horizontal lines) 
e (Class): sophisticated hypothesis G (like black curve) 


e Teacher: a tactic learning algorithm that directs the students to 
focus on key examples 


Diversity by Re-weighting 
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if g; ‘not good’ for u('+1) — gr-like hypotheses not returned as g:.: 
=> 91,1 diverse from gr 
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idea: construct u(t+1) to make g; random-like 
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* one possibility by re-scaling (multiplying) weights, if 
(total u of incorrect) = 1126 ; | (total ul? of correct) = 6211 ; 
(weighted incorrect rate) = 23 | (weighted correct rate) = $211 
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'optimal' re-weighting under weighted incorrect rate e;: 
multiply incorrect œ (1 — er); multiply correct œ et 


Adaptive Boosting Algorithm 
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define scaling factor 4; = Uu 
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e equivalent to optimal re-weighting 
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—physical meaning: scale up incorrect; scale down correct 
—like what Teacher does 
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O obtain g; by A(D, ul”), 
where A tries to minimize u(*)-weighted 0/1 error 


@ update u(? to u+) by 4, = =, 


where c; = weighted error (incorrect) rate of g; 
return G(x) =? 
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* want g; ‘best’ for Ein: y = " 

2 G(x): 
* uniform? but g very bad for Ein (why? :-)) 
* linear, non-linear? as you wish 
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O obtain g; by A(D, ul”), where ... 


© update u(? to ul") by 4, = YE, where ... 
© compute o; = In(4;) 
return G(X) — sign (EL, ax91(X)) 
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* wish: large o; for ‘good’ gt <= a; = monotonic(#;) 
e will take Qt = In(4;) 
© e =3 =>, = 1 = a, = 0 (bad g; zero weight) 
© et = 0 — 4 = œ => art = oo (super g: superior weight) 
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Adaptive Boosting = weak base learning algorithm .4 (Student) 
+ optimal re-weighting factor 4: (Teacher) 
+ ‘magic’ linear aggregation a; (Class) 
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© obtain g; by A(D, ul"), 
where A tries to minimize u(')-weighted 0/1 error 
© update ul to ul") by 
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where @; = ,/—* and e; = 
© compute o; = In(#:) 
return G(x) = sign [x BEACH 
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» From VC bound 
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* first term can be small: 

En(G) = 0 after T = O(log N) iterations if e; < « < } always 
* second term can be small: 

overall dyco grows "slowly" with T 
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boosting view of AdaBoost: 
if A is weak but always slightly better than random (+; < e < 3), 
then (AdaBoost+.A) can be strong (Ein = 0 and E, small) 


Adaptive Boosting in Action 


ERE re 1 S98S" REA (e X ec +, HEUER) , 
MRE IIA En = 0. SM xit zu decision stumpti, FAMA 
— IF, RAAAdaBoost Wa) decision sump ASEGRA. 


DU REIR, "Eet ETC KC. FUE decision stumpfXHJEI, 


initially 








ILES, AURA, HANERHEERTEITF I, MEREEN 
73: 


x 


NS WT rale ARES, SSA sI 
AMZE—TSsiN RATE. WRATH Ada boer, decision stumpElX 
DEl. BRANDEN, BETO REG FAT. 


t = 100 





AAW, AdaBoost-Stump HIER ERERENIDARNIERFA FREI DEN 
Ro 


MEME -—^4AdaBoost-StumpfE A BS HE 75 TRIB NL : 


d 





original picture by F.U.S.LA. assistant and derivative work by Sylenius via Wikimedia Commons 


The World's First ‘Real-Time’ Face Detection Program 





» AdaBoost-Stump as core model: linear aggregation of key 
patches selected out of 162,336 possibilities in 24x24 images 
—feature selection achieved through AdaBoost-Stump 

» modified linear aggregation G to rule out non-face earlier 
—efficiency achieved through modified linear aggregation 
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